Evaluating the Use of a Language Model to Crowdsource Gun Violence Reports
Using AI with crowdsourced social media data is a promising resource for human rights monitoring (Alhelbawy et al, 2020), but…
Keyword-based search leads to a high proportion of unrelated text.
Organizations have limited resources to deal with large volumes of data.
Previous studies have explored machine learning models for human rights monitoring (Alhelbawy et al., 2020; Pilankar et al., 2022; Ta et al., 2022).
However, there was a gap in the literature regarding applications on…
Applications for Portuguese-language texts.
Evaluations of adopting these models in real-world settings.
We deployed an open-source language model to assist crowdsourcing gun violence events with social media.
Partnering with a Brazilian organization allowed us to systematically evaluate its application in a real-world setting in 2023.
Fogo Cruzado (literally “Crossfire”) monitors events of gun violence in four Brazilian cities.
A small team of analysts track social media posts 24/7.
They have been interacting with users who report gun violence on Twitter/X for years
Keyword-based search with geographical filters on Tweetdeck.
RQ1 - Can Transformer-based language models accurately identify gun violence reports in Brazilian Portuguese social media texts?
RQ2 - What are the advantages and challenges of adopting a language model for real-time monitoring compared to manually reviewing social media texts?
RQ1: BERT model trained on past analysts’ interactions with tweets reporting gun violence.
RQ2: We analyzed surveys, interviews, and interaction metrics (with a diff-in-diff model) before and after the adoption of our prototype.
A BERT-based model in Portuguese achieved good performance. Inference can be performed on CPUs.
Tweets were updated every fifteen minutes.
The prototype effectively filtered out less relevant social media content.
Interview with an analyst
“[Now] I do not have to go hunting for tweets.
Sometimes, I missed them [gun violence reports] because there were too many [unrelated] messages. During the BBB [Big Brother Brasil, an annual TV show extremely popular on Twitter], it was chaotic [. . . ]. It was literally a treasure hunt”
Our prototype removes the need for restrictive geolocation filters, allowing analysts to expand their search scope.
We estimated that analysts using the model engaged in nine additional daily interactions with users reporting events.
The interviews and surveys allowed us to identify three major shortcomings:
The delay between each update: promptly updating new tweets is critical.
Static keywords used for search: terms need to be dynamically set to monitore live conflicts.
Use of text-only features: profile images also help analysts to decide if they will interact with users.
Major LLMs may be powerful, but small, open-source models remain useful.
Human rights monitoring: Academics 🤝 NGOs
The closure of social media APIs poses a major obstacle for low-resourced organizations.
Ayman Alhelbawy, Mark Lattimer, Udo Kruschwitz, Chris Fox, and Massimo Poesio. An NLP-Powered Human Rights Monitoring Platform. Expert Systems with Applications, 153, 2020. ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2020.113365.
Yash Pilankar, Rejwanul Haque, Mohammed Hasanuzzaman, Paul Stynes, and Pramod Pathak. Detecting Violation of Human Rights via Social Media. In Proceedings of the First Computing Social Responsibility Workshop within the 13th Language Resources and Evaluation Conference, pages 40–45. European Language Resources Association, 2022. https://aclanthology.org/2022.csrnlp-1.6.
Hoang Thang Ta, Abu Bakar Siddiqur Rahman, Lotfollah Najjar, and Alexander Gelbukh. GAN-BERT: Adversarial Learning for Detection of Aggressive and Violent Incidents from Social Media. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022), CEUR Workshop Proceedings, 2022. https://ceur-ws.org/Vol-3202/davincis-paper7.pdf.
Thank you!
📧 Email: adriano@belisario.website
🌐 Website: belisario.website
👨🏻🏫 Presentation: belisario.website/crossfire_paper/